Maps preserving the nilpotency of products of operators

Let B(X) be the algebra of all bounded linear operators on the Banach space X, and let N (X) be the set of nilpotent operators in B(X). Suppose φ : B(X)→ B(X) is a surjective map such that A,B ∈ B(X) satisfy AB ∈ N (X) if and only if φ(A)φ(B) ∈ N (X). If X is infinite dimensional, then there exists a map f : B(X)→ C \ {0} such that one of the following holds: (a) There is a bijective bounded linear or conjugate-linear operator S : X → X such that φ has the form A 7→ S[f(A)A]S−1. (b) The space X is reflexive, and there exists a bijective bounded linear or conjugate-linear operator S : X ′ → X such that φ has the form A 7→ S[f(A)A′]S−1. If X has dimension n with 3 ≤ n <∞, and B(X) is identified with the algebra Mn of n× n complex matrices, then there exist a map f : Mn → C \ {0}, a field automorphism ξ : C → C, and an invertible S ∈Mn such that φ has one of the following forms: A = [aij ] 7→ f(A)S[ξ(aij)]S or A = [aij ] 7→ f(A)S[ξ(aij)]S, where A denotes the transpose of A. The results are extended to the product of more than two operators and to other types of products on B(X) including the Jordan triple product A ∗B = ABA. Furthermore, the results in the finite dimensional case are used to characterize surjective maps on matrices preserving the spectral radius of products of matrices. AMS classification: 47B49.